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Game theory formalizes certain interactions between physical particles or between living beings in 
biology, sociology, and economics, and quantifies the outcomes by payoffs. The prisoner's dilemma 
(PD) describes situations in which it is profitable if everybody cooperates rather than defects (free- 
rides or cheats), but as cooperation is risky and defection is tempting, the expected outcome is de- 
fection. Nevertheless, some biological and social mechanisms can support cooperation by effectively 
transforming the payoffs. Here, we study the related phase transitions, which can be of first order 
(discontinous) or of second order (continuous), implying a variety of different routes to cooperation. 
After classifying the transitions into cases of equilibrium displacement, equilibrium selection, and 
equilibrium creation, we show that a transition to cooperation may take place even if the stationary 
states and the eigenvalues of the replicator equation for the PD stay unchanged. Our example is 
based on adaptive group pressure, which makes the payoffs dependent on the endogeneous dynamics 
in the population. The resulting bistability can invert the expected outcome in favor of cooperation. 

PACS numbers: 02.50.Lc,87.23.Gc,87.23.Kg,89.65.-s 
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Introduction. Game theory goes back to von Neumann 
[H, one of the superminds of quantum mechanics. Origi- 
nally intended to describe interactions in economics, soci- 
ology, and biology , it has recently become a quickly 
growing research area in physics, where methods from 
non-linear dynamics and pattern formation 0, Q , agent- 
based or particlc-likc models network theory Q 
and statistical physics Q are applied. There are even 
quantum theoretical contributions 0] . 

When two entities characterized by the states, "strate- 
gies", or "behaviors" i and j interact with each other, 
game theory formalizes the result by payoffs Pij , and the 
structure of the payoff matrix {Pij) determines the kind 
of the game. The dynamics of a system of such entities 
is often delineated by the so-called replicator equations 



dp{i,t) 



dt 



= p{i,t) 



(1) 



i 



pii^t) represents the relative frequency of behav- 
ior i in the system, which increases when the expected 
"success" Fi = J2j PijP{jj exceeds the average one. 

Many collective phenomena in physics such as agglom- 
eration or segregation phenomena can be studied in a 
game-theoretical w a,y S B. Apphcations also include the 
theory of evolution [lol| and the study of ecosystems [ll| . 
Another exciting research field is the study of mecha- 
nisms supporting the cooperation between selfish indi- 
viduals [l|-|3| in situations like the "prisoner's dilemma" 
or public goods game, where they would usually defect 
(free-ride or cheat). Contributing to public goods and 
sharing them constitute ubiquitous situations, where co- 
operation is crucial, for example, in order to maintain a 
sustainable use of natural resources or a well-functioning 
health or social security system. 

In the following, we will give an overview of the sta- 



tionary solutions of the replicator equations ([T]) and their 
stability properties. Based on this, we will discuss sev- 
eral "routes to cooperation" , which transform the pris- 
oner's dilemma into other games via different sequences 
of continuous or discontinuous phase transitions. These 
routes will then be connected to different biological or 
social mechanisms accomplishing such phase transitions 
[l^ . Finally, we will introduce the concept of "equilib- 
rium creation" and distinguish it from routes to coop- 
eration based on "equilibrium selection" or "equilibrium 
displacement" . A new cooperation-promoting mechanism 
based on adaptive group pressure will exemplify it. 

Stability properties of different games. Studying games 
with two strategies i only, the replicator equations ([T|) 
simplify, and we remain with 



dpjt) 
dt 



= p{t)[l~p{t)]{X,[l~p{t)]~X,p{t)}, (2) 



where p{t) = p(l, t) represents the fraction of coopcrators 
and 1 — p{t) = p{2,t) the fraction of defectors. Ai ~ 
Pi2 — P22 and A2 — P21 — Pii are the eigenvalues of 
the two stationary solutions p = pi = and p ~ P2 = 
1. If < Ai/(Ai + A2) < 1, there is a third stationary 
solution p = pj, ~ Ai/(Ai + A2) with eigenvalue A3 = 
— (1 — p3)Ai. For the sake of our discussion, we imagine 
an additional fluctuation term ^(t) on the right-hand-side 
of Eq. reflecting small perturbations of the strategy 
distribution. 

Four different cases can be classified 0]: (1) If Ai < 
and A2 > 0, the stationary solution pi corresponding 
to defection by everybody is stable, while the stationary 
solution p2 corresponding to cooperation by everyone is 
unstable. That is, any small perturbation will drive the 
system away from full cooperation towards full defection. 
This situation applies to the prisoner's dilemma (PD) de- 
fined by payoffs with P21 > Pii > P22 > Pi2- According 
to this, strategy i ~ I ("cooperation") is risky, as it can 
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yield the lowest payoff P12, while strategy i = 2 ("de- 
fection" ) is tempting, since it can give the highest payoff 
P21. (2) If Ai > and A2 < 0, the stationary solution pi is 
unstable, while p2 is stable. This means that the system 
will end up with cooperation by everybody. Such a situ- 
ation occurs for the so-called harmony game (HG) with 
Pii > P21 > P12 > P22, as mutual cooperation gives the 
highest payoff Pn. (3) If Ai > and A2 > 0, the station- 
ary solutions pi and p2 are unstable, but there exists a 
third stationary solutionis, which turns out to be stable. 
As a consequence, the system is driven towards a situ- 
ation, where a fraction ps of cooperators is expected to 
coexist with a fraction {1 — ps) of defectors. Such a situ- 
ation occurs for the snowdrift game (SD) (also known as 
hawk-dove or chicken game). This game is characterized 
by P21 > Pii > P12 > P22 and assumes that unilateral 
defection is tempting, as it yields the highest payoff P21, 
but also risky, as mutual defection gives the lowest payoff 
P22. (4) If Ai < and A2 < 0, the stationary solutions pi 
and p2 are both stable, while the stationary solution p^ is 
unstable. As a consequence, full cooperation is possible, 
but not guaranteed. In fact, the final state of the system 
depends on the initial condition p{0) (the "history"): If 
p{0) < P3, the system is expected to end up in the station- 
ary solution pi, i.e. with full defection. If p(0) > pa, the 
system is expected to move towards p2 = 1, correspond- 
ing to cooperation by everybody. The history-dependence 
implies that the system is multistable (here: bistable) , as 
it has several (locally) stable solutions. This case is found 
for the stag hunt game (SH) (also called assurance). This 
game is characterized by Pn > P21 > P22 > P12, i-e. 
cooperation is rewarding, as it gives the highest payoff 
Pn in case of mutual cooperation, but it is also risky, as 
it yields the lowest payoff P12 , if the interaction partner 
is uncooperative. 

Phase transitions and routes to cooperation. When fac- 
ing a prisoner's dilemma, it is of vital interest to trans- 
form the payoffs in such a way that cooperation be- 
tween individuals is supported. Starting with the pay- 
offs P? of a prisoner's dilemma, one can reach differ- 
ent payoffs Pij, for example, by introducing strategy- 



dependent taxes Ty 



PO- 



Pii > 0. When increas- 



ing the taxes from to T?-, the eigenvalues will 
change from A? = P^^ - P°2 and A^ = P^^ - P^^ to 
Ai = A? -I- T22 - P12 and A2 = A2 -t- Tn - T21 . In this way, 
one can create a variety of routes to cooperation, which 
are characterized by different kinds of phase transitions. 
We define route 1 [PD^HG] by a direct transition from 
a prisoner's dilemma to a harmony game. It is charac- 
terized by a discontinuous transition from a system, in 
which defection by everybody is stable, to a system, in 
which cooperation by everybody is stable (see Fig. la). 
Route 2 [PD— 7>SII] is defined by a direct transition from 
the prisoner's dilemma to a stag hunt game. After the 
moment i*, where A2 changes from positive to negative 



values, the system behavior becomes history-dependent: 
When the fluctuations S^{t) for t > t^, exceed the crit- 
ical threshold P3{t) = Ai/[Ai + A2(t)], the system will 
experience a sudden transition to cooperation by every- 
body. Otherwise one will find defection by everyone, as 
in the prisoner's dilemma (see Fig. lb). In order to make 
sure that the perturbations ^(i) will eventually exceed 
P3{t) and trigger cooperation, the value of A2 must be 
reduced to sufficiently large negative values. It is also 
possible to have a continuous rather than sudden tran- 
sition to cooperation: We define route 3 [PD— s-SD] by 
a transition from a prisoner's dilemma to a snowdrift 
game. As Ai is changed from negative to positive val- 
ues, a fraction P3(t) = Ai(t)/[Ai(i) -I- A2] of coopera- 
tors is expected to result (see Fig. Ic). When increas- 
ing Ai, this fraction rises continuously. One may also 
implement more complicated transitions. Route 4, for ex- 
ample, establishes the transition sequence PD^SD^HG 
(see Fig. Id), while we define route 5 by the transition 
PD— J-SH^-HG (see Fig. le). One may also implement 
the transition PD^SD^HG-^SH {route 6, see Fig. If), 
establishing a path-dependence, which can guarantee co- 
operation by everybody in the end. (When using route 2, 
the system remains in a defective state, if the perturba- 
tions do not exceed the critical value p^.) 



(a) 



(b) 



COOP 
HG 



Route 1 




1 






1 




^'Vicoop 






DEF. 






DEF. 


PD 


/:"o 




PD 





Route 4 


> 






(f) 



icoop 

• HG 



p, 
1 




DEF 


™y;coop:«'^I** 


PD 


/ • HG : , 




/ SD : :_/ SH 





Route 6 



Route 5 



FIG. 1: Schematic illustration of the phase transitions defin- 
ing the different routes to cooperation. The order parameter 
is the stationary frequency of cooperators, while the control 
parameters are the parameters r, w, k, m, or q in Nowak?s 
cooperation-enhancing rules [l2| (see main text) or, more gen- 
erally, (non-)linear combination of the model parameters b 
and c. Solid (red) lines represent stable stationary proportions 
of cooperators, dashed lines unstable fix points. Diagonal lines 
show the additional stationary solution pa, where < pa < 1. 
(p = proportion of cooperators; DEFECT = defection is sta- 
ble, i.e. everybody defects; COOP = cooperation is stable, i.e. 
everybody cooperates; COEX = mixture of defectors with a 
proportion pa of cooperators; BISTAB = cooperation is sta- 
ble if P3 < p(0), where p(0) means the initial proportion of 
cooperators, otherwise everybody defects.) 



Relationship with cooperation- supporting mechanisms. 
We will now discuss the relationship of the above intro- 
duced routes to cooperation with biological and social 
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mechanisms ("rules") promoting the evolution of coop- 
eration. Martin A. Nowak performs his analysis of five 
such rules with the reasonable specifications T = b > 0, 
R = b- c>0, S=~c<0, and P = in the limit of 
weak selection [l^]- Cooperation is assumed to require a 
contribution c > and to produce a benefit 6 > c for the 
interaction partner, while defection generates no payoff 
(P = 0). As most mechanisms leave Ai or A = (Ai-|-A2)/2 
unchanged, we will now focus on the payoff-dependent 
parameters Ai and A (rather than Ai and A2). The ba- 
sic prisoner's dilemma is characterized by A^ = — c and 
AO =0. 

According to the Supporting Online Material of Ref. 
(l^ . kin selection (genetic relatedness) tranforms the 
payoffs into Pn = P^^ + r{b - c), Pu = P^^ + br, 
P21 = P21 — cr, and P22 — P22- Therefore, it leaves A 
unchanged and increases Ai by r22 — ^12 = iir, where r 
represents the degree of genetic relatedness. Direct reci- 
procity (repeated interaction) does not change Ai, but 
it reduces A by —\[b — c)[l/(l — w) — 1] < 0, where w 
is the probability of a future interaction. Network reci- 
procity (clustering of individuals playing the same strat- 
egy) leaves A unchanged and increases Ai by H{k), where 
H(k) is a function of the number k of neighbors. Finally, 
group selection (competition between different popula- 
tions) increases Ai by (6 — c)(m — 1), where m is the 
number of groups, while A is not modified. However, Ai 
and A may also change simultaneously. For example, indi- 
rect reciprocity (based on trust and reputation) increases 
Ai by eg and reduces A by — 1(6 — c)^ < 0, where q quan- 
tifies social acquaintanceship. 

Summarizing this, kin selection, network reciprocity, 
and group selection preserve A = and increase the value 
of Ai (see route 1 in Fig. 2). Direct reciprocity, in con- 
trast, preserves the value of Ai and reduces A (see route 
2a in Fig. 2). Indirect reciprocity promotes the same tran- 
sition (see route 2b in Fig. 2). Supplementary, one can 
analyze costly punishment. Using the payoff specifications 
made in the Supporting Information of Ref. [31 , costly 
punishment changes A by — (^ -|- 7)/2 < and Ai by 
—7 1J|, i.e. when 7 is increased, the values of A and Ai 
are simultaneously reduced (see route 2c in Fig. 2). Here, 
7 > represents the punishment cost invested by a co- 
operator to impose a punishmet fine /3 > on a defector, 
which decreases the payoffs of both interaction partners. 
Route 3 can be generated by the formation of friendship 
networks [Isj . Route 4 may occur by kin selection, net- 
work reciprocity, or group selection, when starting with 
a prisoner's dilemma with A" < (rather than A^ = 
as assumed before). Route 5 may be generated by the 
same mechanisms, if A^ > 0. Finally, route 6 can be im- 
plemented by time-dependent taxation (see Fig. 2). 

Further kinds of transitions to cooperation. The routes 
to cooperation discussed so far change the eigenvalues Ai 
and A2 , and leave the stationary solutions pi and p2 un- 
changed. However, transitions to cooperation can also be 




BISTAB @ 

SH 



FIG. 2: Phase diagram of expected system behaviors, based on 
an analysis of the game-dynamical replicator equation (2) as 
a function of the parameters A and Ai . The different routes to 
cooperation are illustrated by arrows. Terms in capital letters 
are defined in Fig. f . Inset: Stable stationary solutions (solid 
lines) and unstable ones (broken lines) as functions of the 
parameter K, when the reward depends on the proportion of 
cooperators. The bifurcation at the "tipping point" K = Ko 
"inverts" the system behavior (see main text). 



generated by shifting the stationary solutions or creating 
new ones, as we will show now. For this, we generalize the 
replicator equation ^ by replacing Ai with f{p) and A 
with g(jp), and by adding a term h{p), which can describe 
effects of spontaneous transitions like mutations. To guar- 
antee < p[t) < 1, we must have h{p) = v{p) — pw{p) 
with functions w(j)) > v(p) > 0. The resulting equa- 
tion is dp/dt = F{p{t)) with F{p) = (I - p)[f{p) - 
2g{p)p]p + h{p), and its stationary solutions pk are given 
by P(pfc) = {l^pk)[fipk)-2g{pk)Pk]Pk + h{pk) = 0. The 
associated eigenvalues A^. = dF(jpk)/dp determining the 
stability of the stationary solutions pk are 

Afe = {l-2pk)ifk-2pkgk)+Pk{l-Pk){.f'k-'2pkg'k-'29k)+h'k. 



where we have used the abbreviations fk = f{pk), Qk = 
gijpk), hk = h{pk). fk = f'iPk), g'k = g'iPk) and hk = 
h'[pk) are the derivatives of the functions /(p), g{p) and 
h{p) in the points p — pk. 

Classification. We can now distinguish different kinds 
of transitions from defection to cooperation: If the sta- 
tionary solutions pi ~ Q and p2 = 1 of the prisoner's 
dilemma are modified, we talk about transitions to coop- 
eration by equilibrium displacement. This case occurs, for 
example, when random mutations are not weak (h ^ Q). 
If the eigenvalues Ai or A2 of the stationary solutions 
Pi = and P2 = f are changed, we speak of equilib- 
rium selection. This case applies to all routes to coop- 
eration discussed before. If a new stationary solution 
appears, we speak of equilibrium creation. The differ- 
ent cases often appear in combination with each other 
(see the Summary below). In the following, we will dis- 
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cuss an interesting case, where cooperation occurs solely 
through equilibrium creation, i.e. the stationary solutions 
Pi and p2 of the replicator equation for the prisoner's 
dilemma as well as their eigenvalues Ai and A2 remain un- 
changed. We illustrate this by the example of an adaptive 
kind of group pressure that rewards mutual cooperation 
(Til < 0) or sanctions unilateral defection (T21 > 0). 
Both, rewarding and sanctioning reduces the value of 
A2, while Ai remains unchanged. Assuming here that the 
group pressure vanishes, when everybody cooperates (as 
it is not needed then), while it is maximum when every- 
body defects (to encourage cooperation) [l5| , wc may set 
f{p) = Ai and g{p) = A° — K[l — p{t)], corresponding 
to A2(i) = A^ - 2K[l - p{t)]. It is obvious that we still 
have the two stationary solutions pi ^ and P2 — ^ with 
the eigenvalues Ai = A? < and A2 = 2A° - A? > of 
the original prisoners dilemma with parameters Ai and 
A2 or A°. However, for large enough values of K [namely 
for K > Ko = X° + \X'i\ + \/|A^|(2A0 + |A'i'|)], we find two 
additional stationary solutions 

- i - ^ ± ,/(! - - ^ (3) 

P- is an unstable stationary solution with pi < p- < p+ 
and A_ = dF{p^)/dp > 0, while is a stable stationary 
solution with p_ < < p2 and A-(. — dF{p+) / dp < 
(see inset of Fig. 2). Hence, the assumed dependence of 
the payoffs on the proportion p of cooperators generates 
a bistable situation (BISTAB), with the possibility of a 
coexistence of a few defectors with a large proportion 
p+ of cooperators, given K > Kq- If p{0) < p-, where 
p(0) denotes the initial condition, defection by everybody 
results, while a stationary proportion p+ of cooperators 
is established for p_ < p(0) < 1. Surprisingly, in the 
limit if — >■ 00, cooperation is established for any initial 
condition p(0) ^ (or through fluctuations). 

Summary. We have discussed from a physical point of 
view what must happen that social or biological, payoff- 
changing interaction mechanisms can create cooperation 
in the prisoner's dilemma. The possible ways are (i) mov- 
ing the stable stationary solution away from pure de- 
fection (routes 3, 4, and 6), (ii) stabilizing the unstable 
solution (routes 1, 2, 4, 5 and 6), or (iii) creating new 
stationary solutions, which are stable (routes 3, 4 and 6). 
Several of these points can be combined. If (i) is fulfilled, 
we speak of "equilibrium displacement" , if their eigenval- 
ues change, wc called this "equilbrium selection" , and if 
(iii) is the case, wc talk of "equilibrium creation". The 
first case can result from mutations, the second one ap- 
plies to man y so cial or biological cooperation-enhancing 
mechanisms [I4I . We have discussed an interesting case of 
equilibrium creation, in which the outcome of the replica- 
tor equation is changed, although the stationary solutions 
of the PD and their eigenvalues remain unchanged. This 
can, for example, occur by adaptive group pressure [l5l |. 
which introduces an adaptive feedback mechanism and 



thereby increases the order of non-linearity of the repli- 
cator equation. Surprisingly, already a linear dependence 
of the payoff values Pij on the endogeneous dynamics 
p{t) of the system is enough to destabilize defection and 
stabilize cooperation, thereby inverting the outcome of 
the prisoner's dilemma. 
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